[None][feat] BREAKING: Support Inkling-NVFP4 model - #17062

…(6*448))

Root cause of the baseline NVFP4 accuracy gap vs SGLang. The Inkling checkpoint
ships routed-expert activation calibration as a RAW `.input_amax`, but the
fused-MoE loader / trtllm `fp4_quantize` expect the ModelOpt per-tensor
`input_scale = amax / (E2M1_MAX * E4M3_MAX) = amax / (6*448)`. The mapper renamed
`.input_amax` -> `input_scale` WITHOUT the conversion, making the activation
global scale 2688x too small: every routed expert's fp4 output was ~0.62 rel_rms
from the bf16 ground truth (7.6x SGLang's 0.082). Mirrors sglang inkling.py:1222 /
inkling_common/dense_mlp.py:497. Positional bisection confirmed the
weight/block-scale/gate-up-interleave layout was already element-wise correct
(24/24 L3 experts, 18.8M elems, 0 diverged) -- the defect was ONLY the activation
input scale, shared by both TRT MoE backends.

Fix: `inkling_weight_mapper._map_expert` divides `.input_amax` by (6*448).

Result (baseline: cuda_graph=off, overlap=off, TP=4, CUTLASS MoE, fp4 fix active):
- L3 routed-expert vs bf16 truth: rel_rms 0.624 -> 0.081 (== SGLang 0.082)
- GSM8K paired 5x100:  TRT 0.916 -> 0.968 vs SGLang 0.974; mean_delta -0.056 ->
  -0.006 (Gate 2 PASS, within +/-0.02); runaway canary passes; collapse 0
- MMLU paired 6x100:   TRT ~0.822 -> 0.862 vs SGLang 0.872; gap -3.44pt -> -1.0pt
  (Gate 3 accuracy PASS); B-bias error-is-B 65.9% -> 40%, TRT-B 33.2% -> 29.8%

Residual (not this fix): TRT (CUTLASS) vs SGLang (flashinfer) fp4 kernel-family
near-tie noise -- accuracy-neutral, not bit-reproducible at batch>1. The strict
within-2pp-of-gold MMLU B-bias criterion is model-inherent (SGLang itself
over-picks B) and is left to human adjudication, not a TRT defect.

Also adds env-gated per-layer/per-module dump instrumentation
(modeling_inkling.py dump_sink; inkling_perlayer_localize_test.py) and the
trtllm-gen MoE backend path used to localize the bug.

Signed-off-by: kleinc <kleinc@nvidia.com>

…ring graph capture

Root cause of the Inkling TP=4 enabled-runtime (cuda_graph=on + overlap) decode collapse
("B2"): the AutoTuner-selected all-reduce (`tunable_allreduce`, AUTO strategy) is not
CUDA-graph-capture-safe. Frozen into a decode graph, the tuned tactic produces a non-finite
result on replay, so decode goes NaN from the first global-attention layer and collapses to
a token-0 repeat ("Paris!!!!"). Eager is finite at identical metadata; the fault is baked
into the captured graph.

Localized by single-variable determinism isolation (autotuner ON -> collapse, autotuner OFF
-> clean), NOT by op-fingerprints -- any in-graph probe shifts the graph memory pool / tactic
selection and suppresses the bug (a Heisenbug).

Fix (`tensorrt_llm/_torch/distributed/ops.py`, AllReduce.forward): during CUDA-graph capture
(`torch.cuda.is_current_stream_capturing()`), skip `tunable_allreduce` and fall back to the
static, graph-safe `all_reduce_op(AUTO)`. Warm-up / eager (not capturing) still autotune, so
steady-state performance is unchanged. This is a mainline TRT-LLM robustness fix (any TP
model that captures an AUTO all-reduce benefits), not Inkling-specific.

Confirmed: enabled generation_parity is BIT-IDENTICAL to the baseline (cuda_graph=off) --
tf_mismatch/neartie/confident=16/11/5, freerun_collapse=0, identical logit_checksum; enabled
5x100 GSM8K (0.966 vs SGLang 0.974) and MMLU (0.875 vs 0.872) within +/-0.02, zero errors,
no regression vs the accepted CUTLASS baseline.

Also includes env-gated per-layer/per-op B2 localization instrumentation
(`modeling_inkling.py` dump_sink / INKLING_FP*; `inkling_fp_localize_test.py`), zero-cost
when the INKLING_FP* env is unset.

Signed-off-by: kleinc <kleinc@nvidia.com>

…MLP tower, image fusion (WIP)

WIP -- Stage-1 progress snapshot on the Inkling NVFP4 multimodal tower, on top
of the accepted text tower. Not finished: the vision path is verified clean but
the MMMU Accounting gap is still open (decode-side), and audio / MTP are still
deferred. Committed to record current progress, not as a complete feature.
Text decode/attention/MoE paths are untouched.

Model / config:
- configs/inkling.py: add image_token_id (200054, the in-vocab chat-template
  <|unused_200054|>) and audio_token_id. The SGLang-internal -101 sentinel is
  rejected by TensorRT-LLM's executor token-id validation; the two ids are
  interchangeable for parity since both are overwritten by vision embeddings.
- modeling_inkling_vision.py (new): hMLP vision tower InklingVisionModel and
  InklingInputProcessor, which expands the <image> placeholder to one token per
  vision patch and attaches vision_patches_bthwc features.
- modeling_inkling.py: InklingForConditionalGeneration registers the input
  processor, builds the vision tower as a replicated bf16 submodule, and fuses
  per-patch embeddings into the text stream via fuse_input_embeds with explicit
  text/mm indices (OOV-safe). Fixes the "image not visible" hallucination: the
  fused stream must NOT be re-normed, since SGLang scatters raw vision rows in
  after embed_norm; the extra RMSNorm corrupted the image rows.

Tests / diagnostics (tests/unittest/_torch/modeling/, 25 new files): MMMU
harness alignment against the SGLang scorer, input-processor and vision-tower
unit checks, image e2e / fusion / logit-replay / generation-parity drivers, and
the localization probes used to isolate the vision-vs-decode split (vision
verified bitwise-clean; the residual Accounting gap is decode-side). These
require GPU + the NVFP4 checkpoint (TP=4) and were not run for this commit.

Signed-off-by: kleinc <kleinc@nvidia.com>

…s_token

SamplingParams._setup() looked for the end-of-generation token in only two
places: tokenizer.eos_token_id, then generation_config.eos_token_id. When a
checkpoint provides neither, end_id stayed None, nothing could terminate a
request, and every generation silently ran to max_tokens.

That combination is reachable. Multi-part chat formats have no single
terminator -- a message end, an end-of-sampling marker and a document separator
are distinct tokens -- so such checkpoints register their control tokens under
extra_special_tokens / additional_special_tokens, neither of which populates
tokenizer.eos_token_id, and declare the real stop token as eos_token_id in
config.json. Those checkpoints also tend to ship no generation_config.json.
_setup() already received hf_model_config but never consulted it.

Observed on a checkpoint of that shape: responses ran to the token limit while
emitting the configured eos_token_id up to 441 times in a single response.

Fall back to hf_model_config.eos_token_id when the first two sources yield
nothing. A list value sets end_id from its first entry and appends the rest to
stop_token_ids, mirroring the existing generation_config path. Priority is
otherwise unchanged: an explicit SamplingParams(end_id=...) still wins and
tokenizer.eos_token_id still takes precedence over config.json, so models that
already resolved an end_id see no behavioural change.

Warn when all three sources come up empty. Generating to the cap on every
request with no diagnostic is the part that makes this expensive to find.

Add unit tests for tokenizer priority, the config fallback, the list form, an
explicit end_id not being overridden, the all-empty case, and a missing
hf_model_config.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

`logits_rows` is a view of the same slice that is written back, so the
assignment self-overlaps and torch rejects it with "... refer to a single
memory location. Please clone()...". The processors edit in place, so the
write-back is redundant anyway; cloning the source makes it overlap-safe.

Only reached for requests carrying a py_logits_post_processor, so normal
requests are unaffected.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

… parity runs

evaluate/interface.py gains summarize_generation_stats/log_generation_stats,
and lm_eval emits a greppable GEN_STATS marker per batch. Without it a
runaway / no-EOS regression hides behind a parseable answer buried in a wall
of repeated text -- the failure mode that cost several bring-up iterations.

The MMMU harness/runner changes carry the sharded union runs used for the
TRT-vs-SGLang comparison (shard plan, incremental atomic per-item writes so a
wall-killed shard keeps everything it scored, cap/max_seq plumbing), plus unit
coverage of the answer parser.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Adds streaming-vs-batch equivalence over arbitrary split points, tool-call and
repetition segmentation, and end-tokens split across deltas.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Standalone probes used to localize the TRT-vs-SGLang vision divergence:
prefill logits, transformers-reference decode, termination behaviour, and a
per-layer activation dump. Diagnostics only -- not part of any test suite.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…del code

The cuda-graph decode collapse is a defect in symmetric all-reduce: a captured
NCCL_SYMMETRIC reduce whose send buffer is unregistered while its recv buffer is
a registered NCCL window corrupts the run at a 12288 B message. Inkling hits
that size exactly -- hidden 6144, bf16, one decode token -- so the first
global-attention layer goes non-finite and decode collapses to a repeated
token 0.

Revert the shared-code mitigation in distributed/ops.py: that file is now
byte-identical to its pre-Inkling state, so no other model's all-reduce path
changes on our account.

Mitigate in modeling_inkling.py instead. The all-reduces that trigger this are
built by generic modules -- attention o_proj, MoE down_proj -- so the strategy
cannot be passed at construction without editing shared code; rebuilding each
AllReduce after super().__init__() keeps the mitigation model-local. Each
rebuilt instance carries the module's own mapping and dtype over, so strategy is
the only delta. Pinning ONESHOT also drops the window requirement, since
AllReduce only takes an NCCL window under NCCL_SYMMETRIC/NCCL/AUTO -- two of the
five trigger conditions go away, not just one.

Active by default; INKLING_ALLREDUCE_STRATEGY=AUTO restores stock behaviour, and
the defect with it, for A/B runs. Measured on job 5728192, alternating arms in
one job against one binary: default 0/3 collapse, AUTO 3/3, 331 modules swapped.

Cost: symmetric is disabled on every Inkling all-reduce, eager included, and
roughly a third of captured decode all-reduces pick it today. The performance
impact is unmeasured and should be measured before this is treated as final.
This is containment, not a root-cause fix -- the defect remains for any other
model that meets all five conditions.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…ences

MMMU already had a reference entry; the two text benchmarks the bring-up
actually measured did not, so the numbers lived only in run logs.

Both references are the cached SGLang NVFP4 measurement under the same
harness, matching how the MMMU entry was sourced:
- GSM8K 95.53 (full set, 0.9553). TRT-Inkling was validated against it with
  the paired 5x100 protocol: flexible-parse mean 0.968, and all four
  cuda-graph x overlap-scheduler corners scored 0.98-0.99 on the 100 paired
  items where SGLang scored 0.99.
- MMLU 85.66 (full Hendrycks, 14042 samples, weighted_accuracy 85.6573). The
  TRT side was measured with the 5-seed text-regression protocol on the
  harness' 114-item subset (83.33 / 84.21 / 85.09 / 85.96 / 87.72, mean
  ~85.3). The comment says so explicitly: it tracks the reference, but no
  full-set TRT-LLM MMLU run has been recorded yet.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…that matters

The bring-up left 22 Inkling files under tests/unittest/_torch/modeling. Most
were per-defect regressions or SGLang-alignment scaffolding whose outcome is
now covered end-to-end by the MMMU/GSM8K accuracy tests. Keep one test per
thing that nothing else covers, drop the rest.

Kept:
- test_modeling_inkling.py       config parse, layer classification, weight
                                 accounting over the real checkpoint
- inkling_vision_tower_test.py   vision tower
- inkling_audio_tower_test.py    audio tower  (no accuracy benchmark covers it)
- inkling_video_utils_test.py    video utils  (no accuracy benchmark covers it)
- inkling_input_processor_test.py multimodal input processor
- inkling_moe_backend_select_test.py  the INKLING_MOE_BACKEND kernel select

inkling_mmmu_real_align_test.py was doing double duty: the vision-tower and
input-processor tests import its MMMU item fetch/cache and its importlib
loader. Split that half out as inkling_mmmu_fixtures.py (a fixture module, no
tests) and drop the SGLang-alignment machinery with the rest.

Deleted (16): the mmmu align/harness/run/parser set, image_prompts, the three
per-modality e2e smokes, image_fusion, image_norm_fix, attn_decode_meta,
gate_up_deinterleave, kv_manager_v2, generation_parity and
source_logit_replay.

Every kept file compiles and no reference to a deleted module remains in the
tree.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

The multimodal file already carried TestInkling_NVFP4::test_nvfp4 for MMMU,
so the vision path had an integration test but the shared text decoder --
what GSM8K and MMLU actually exercise -- had none.

Adds TestInkling_NVFP4 to test_llm_api_pytorch.py running both text
benchmarks against the references recorded earlier. It mirrors the multimodal
class: NVFP4 assert, 16384-token budget for the long chain of thought, and
extract_inkling_content as the post-processor so the <|content_thinking|>
channel is dropped and only the visible answer is scored.

Registered the new id in both lists that already carry the multimodal one:
test-db/l0_b200.yml and qa/llm_function_core.txt.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…keep CUTLASS

The Inkling routed experts now run only the default CUTLASS MoE backend. The
trtllm-gen blockScaleMoe path was an opt-in experiment behind
INKLING_MOE_BACKEND=TRTLLM, never the default, and it needed a dedicated
routing enum plumbed all the way into the CUDA runner to work.

Restored to their pre-bring-up state (the additions were Inkling-only, so
these files are now byte-identical to 44f0521):
- cpp/.../trtllmGenKernels/blockScaleMoe/runner.h  the InklingSinkRenorm = 9
  enum value and its name case
- cpp/.../trtllmGenKernels/blockScaleMoe/runner.cu the precomputed-routing
  dispatch branch
- _torch/modules/fused_moe/routing.py              the matching Python enum
  value and its autotuner-dummy mapping

Removed from the model:
- _inkling_trtllm_moe_backend / _moe_config_with_trtllm_backend and the
  frozen-config copy they needed to retarget moe_backend
- InklingMoeRoutingMethod._trtllm_backend, so routing_method_type is always
  Unspecified and requires_separated_routing goes back to the default
- the per-layer INKLING_MOE_SELECT backend-introspection log

Also deletes inkling_moe_backend_select_test.py, which existed only to cover
that knob, and the stale comment that pointed at it as the fix for the fused
combine's cross-row non-determinism.

INKLING_ALLREDUCE_STRATEGY is now the only environment variable the model
reads. Everything compiles and git diff --check is clean.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

The MMLU comment added in 5f28628 claimed no full-set TRT-LLM run had been
recorded. That is wrong. One exists: 2026-07-20, all 14042 Hendrycks samples,
weighted_accuracy 82.2176 -- 3.44 points below the 85.66 reference, and it
failed the bring-up's 2-point gate. The gap was later traced to an fp4
expert-GEMM family difference (CUTLASS/trtllm-gen vs SGLang's flashinfer) plus
non-terminating generation on hard prompts.

What is true is narrower: the full set has not been re-measured since those
fixes. Post-fix MMLU evidence is subset-only -- a 570-item stratified canary at
84.21 (-1.45, inside the gate) and a 5-seed 114-item regression averaging
~85.3. The comment now says exactly that, including that meeting 85.66 at full
scale is unverified.

GSM8K's comment was not wrong but was too vague about coverage. TRT-LLM has
never been measured on the full 1319-item set: the evidence is the paired
5x100 protocol (flexible mean 0.968), the four cuda-graph x overlap corners at
0.98-0.99 on 100 paired items, and one early full-set attempt that completed
only its first 120-item chunk (0.925 vs 0.975). Spelled out.

Both files still record the SGLang reference as the accuracy value, so
TestInkling_NVFP4::test_nvfp4 grades against a bar the model has not been
shown to clear at full scale.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…timodal

The module already carried the image, audio and video paths, so the "vision"
name no longer described it. Rename it and reorganize the contents into four
labelled sections -- vision tower, audio tower, video tower, and the shared
multimodal input processor -- so each modality is easy to locate.

Behavior-preserving. Along the way:
  * factor the duplicated tower weight-loading into ``_load_tower_weights``
    and the duplicated RMSNorm into a single shared ``InklingRMSNorm``
    (was ``InklingVisionRMSNorm``, used by both towers);
  * factor the input processor's repeated media-list coercion and its
    placeholder/feature-row count checks into small helpers, dropping a
    tautological per-item check (``num_tokens`` is built from ``num_patches``);
  * trim the docstrings to what the code needs, dropping the development-time
    stage/goal references and reference-implementation file paths.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Bring-up leftovers that do not belong in the production model:

  * cuda_graph_runner: drop the "[cuda-graph] CAPTURED/REPLAYED" evidence
    logging and its once-only gate flag. This was pure instrumentation added
    to prove the runtime really captured and replayed a graph, and it sat in
    shared (non-Inkling) code.
  * modeling_inkling: drop the INKLING_ALLREDUCE_STRATEGY environment knob.
    The ONESHOT pin is a correctness mitigation, not a tuning option, so it
    is applied unconditionally instead of via an A/B toggle (and the log line
    that reported it is gone).
  * modeling_inkling: drop the explicit-window short-conv decode path
    (InklingShortConv.forward_decode, the decoder layer's third branch, and
    the attention's conv_states/return_conv_state arguments). Only the
    bring-up replay harness ever passed those; the runtime always drives the
    short convs through the per-request state pool.
  * modeling_inkling: drop the attention's decode_seq_lens / decode_page_table
    / skip_kv_write arguments. The runtime publishes decode metadata into the
    layer's stable GPU buffers before capture, so the pre-supplied static
    tensors had no caller left; the eager fallback that builds them from the
    host block table stays.
  * modeling_inkling: drop InklingConvStateCache.reset (unused) and merge
    InklingConvRuntime.from_metadata into build (the split existed only so the
    replay harness could publish slots itself).

Also rewrite the comments and docstrings across the Inkling _torch files to
describe the code as it stands: no development stage/goal numbers, job ids,
absolute paths into local reference checkouts, or references to the deleted
replay harness. The stale "audio / vision / MTP are deferred" module docstring
now reflects that only MTP is unimplemented.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

The Inkling unit tests were five files carrying a bring-up harness rather than a
test suite: they loaded the SGLang reference from an absolute path in a local
checkout, downloaded MMMU rows over the network into a gitignored cache, wrote
JSON artifacts, hardcoded a personal /lustre checkpoint path, and shipped
__main__ runners that printed comparison tables. None of that can run in CI.

Fold everything multimodal into test_modeling_inkling_multimodal.py -- one file
covering the vision, audio and video paths plus the shared input processor --
with small synthetic configs and inputs: no checkpoint, no GPU, no network, no
SGLang import, no artifacts. 27 tests, well under a second.

Slim test_modeling_inkling.py the same way. The config and layer-classification
tests now build their config explicitly instead of requiring the checkpoint, so
they actually run; the weight-accounting and tensor-shape tests keep the
checkpoint (index JSON only, no weights) and resolve it through the standard
llm_models_root() with an INKLING_CHECKPOINT override, so they skip cleanly
instead of always skipping on a path that only existed on one machine.

Removed:
  inkling_vision_tower_test.py, inkling_input_processor_test.py,
  inkling_audio_tower_test.py, inkling_video_utils_test.py (folded in)
  inkling_mmmu_fixtures.py (MMMU downloader + SGLang loader; no longer used)

Also drop the .gitignore entries for the deleted caches and artifacts.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…nkling PR

Run the repo's pre-commit formatters (isort, yapf, ruff, ruff-format) over the
files this PR touches, and fix the eleven ruff-legacy D205/D209 docstring
regressions it flags in tensorrt_llm/evaluate/interface.py and
tests/unittest/llmapi/test_reasoning_parser.py.

Formatting and docstring wording only; no behavior change.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Two bring-up leftovers outside _torch:

  * evaluate: drop summarize_generation_stats / log_generation_stats and both
    call sites (Evaluator.evaluate and LmEvalWrapper). This logged a greppable
    GEN_STATS line with the finish_reason mix and generated-token distribution
    so a runaway/no-EOS regression would be visible while bringing the model up.
    It is observability scaffolding, not part of the eval contract: it never
    influenced what was scored, nothing consumed the marker, and no test
    covered it.

  * docs: drop "The text decoder is also usable standalone (text-only) via the
    InklingForCausalLM architecture" from the Inkling footnote. It is not true.
    InklingForCausalLM carries no @register_auto_model, so it is not in
    MODEL_CLASS_MAPPING and no checkpoint can select it; the config registry has
    no inkling_text entry either. The class is the base that
    InklingForConditionalGeneration derives from, and the inkling_text handling
    in config_utils/_util exists for that nested sub-config, not for standalone
    loading. The published checkpoint declares InklingForConditionalGeneration
    and the text-only accuracy test loads it through that same architecture, so
    registering the class would advertise a path with no checkpoint to exercise
    it and no test coverage.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…urce gates

CodeRabbit, unresolved: TestInkling_NVFP4.test_nvfp4 defaulted to TP=1 with no
MPI or device-memory gate. Inkling-NVFP4 does not fit on one Blackwell GPU at
free_gpu_memory_fraction=0.6, so the test could not run -- while being listed
in tests/integration/test_lists/test-db/l0_b200.yml and the QA function list.
Coverage that looks present and is not is worse than no coverage.

The review flagged the multimodal test. The text-side TestInkling_NVFP4 in
test_llm_api_pytorch.py has the identical defect -- same checkpoint, same KV
fraction, same missing gates -- so both are fixed here.

Both now request tensor_parallel_size=4 and carry skip_less_mpi_world_size(4)
plus skip_less_device_memory(183000), matching the idiom the other multi-GPU
tests in these files use. TP=4 also matches how the cached references were
measured; at any other parallelism the numbers are not comparable, and the
docstrings now say so.

Also fixes a second unresolved finding in the same function: sampling_params
is a class attribute and AccuracyTask.evaluate sets truncate_prompt_tokens on
it in place when it is None. The loop handed the same object to GSM8K first,
so it kept GSM8K's input budget when MMLU ran and MMLU never applied its own.
Each task now gets its own copy.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…through

CodeRabbit, unresolved: extract_inkling_content only tested for
<|content_text|> / <|content_thinking|> before returning the text unchanged.
Output that carried framing but never opened a visible block -- a
<|message_model|> header with no later content marker, or a generation
truncated right after the header -- therefore came back verbatim, and the
evaluator scored raw special tokens, or text sitting outside any visible
block, as if it were the model's answer.

InklingReasoningParser treats that output as framing and drops it, so the
offline path has to as well. Test the whole control-token set instead, via
_INK_CONTROL_RE, which is the same set the extraction loop already walks.

Passthrough for every other model and benchmark is unchanged: text carrying no
Inkling control token at all still returns unchanged, now on a strictly wider
test.

Three regression tests: framing-only output, truncated-after-header output,
and the unchanged passthrough.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

CodeRabbit, unresolved: when eos_token_id comes from config.json as a list,
the first entry becomes end_id and the rest become stop tokens -- with no check
that the rest are not the primary EOS. A config listing it twice, e.g.
[7, 7, 8], put 7 in both end_id and stop_token_ids. The generation_config path
a few lines below already skips self.end_id; this path did not.

Two regression tests: the repeated-primary case, and a single-element list
producing no stop tokens at all.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…d test helpers

Three unresolved CodeRabbit findings, all the same kind, so they land together
rather than as three near-identical commits.

inkling_weight_mapper.py: _text_config had no return annotation,
preprocess_weights and _map_expert used unparameterized Dict, and the nested
_assign was unannotated. The file already carries `from __future__ import
annotations`, so these use built-in generics (dict[str, torch.Tensor],
re.Match[str]) per the coding guidelines. An AST sweep confirms no function in
the file is left with a missing return or parameter annotation.

reasoning_parser.py: _emit and _consume took bare `list` accumulators and the
three parser entry points declared bare `list` locals, all of which only ever
hold str. Now list[str] throughout.

test_modeling_inkling.py: the checkpoint helpers now return dict[str, str],
set[str] and tuple[list[int], str].

No behaviour change.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Under attention DP every rank runs the FULL attention over its OWN requests and
only the routed experts stay sharded. The base Attention already scopes itself
to that (modules/attention.py builds qkv_proj / o_proj from an internal
tp_size=1 mapping), KVCacheManagerV2 already sizes the paged pool the same way,
Embedding forces tensor_parallel_mode=None, and lm_head is replicated by
DecoderModelForCausalLM. What was missing was every Inkling-only tensor that
hangs off the same split and still read the GLOBAL mapping.tp_size:

  * r_proj sharded num_heads * d_rel by tp_size while the base kept full heads
  * the k/v short convs took tp_shard=True unconditionally, so they convolved a
    quarter of a full-width k/v stream
  * local_num_heads divided by the global tp_size
  * the conv-state pool sized its k/v rows by the global tp_size

None of these fail loudly. They produce a rank that disagrees with its own qkv
projection about how many heads it owns.

InklingDenseMLP (layers 0/1) is a correctness fix rather than a partitioning
choice: its row-parallel down_proj all-reduces a partial sum across the TP
group, and under ADP the peers' partials belong to DIFFERENT requests, so the
reduce would add unrelated tokens together. It now replicates, matching
DeepSeek-V3's _compute_mlp_tp_size, which returns 1 under ADP for this reason.

InklingMoE needs no change: FusedMoE.reducescatter_or_allreduce tests use_dp
before reduce_results, so the reduce_results=True the experts are built with is
correctly superseded by a reduce-scatter under ADP. A test pins that upstream
contract, since Inkling's correctness depends on it with nothing local to blame
if it changed.

__init__ cross-checks its head count against the base rather than trusting two
copies of the attention-TP rule to stay in step.

Tests: 88 pass. Each fix was verified by re-injecting the bug it removes and
confirming the matching test goes from passing to FAILING (not erroring) --
attention head/channel widths, dense-MLP replication, and conv-pool sizing all
caught. Real modules are constructed on a miniature config with the collective
stubbed, because AllReduce's constructor validates the MPI world size and would
otherwise refuse a 4-rank Linear in a single-process pytest.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

The entry said "Attention data parallelism is untested". It is now tested, and
the two things a user needs before enabling it are both non-obvious.

It costs memory rather than saving it. Attention, the embedding, the LM head
and the two dense MLP layers replicate under ADP, so what was a 1/tp slice is a
full copy per rank: measured 145.98 -> 162.68 GB of weights per rank at TP=4.
On a 184 GiB device that leaves little for the KV cache, and the two knobs pull
in OPPOSITE directions -- max_num_tokens / max_batch_size bound the KV-cache
estimation forward, which OOMs before the pool is sized (so
free_gpu_memory_fraction cannot help it), while free_gpu_memory_fraction is
what keeps the pool from starving the scheduler. Turning both down, the
intuitive move, trades an OOM for a scheduler deadlock.

It is also not bit-identical to pure TP, and cannot be: under TP the attention
output is an all-reduced sum of four partials, under ADP it is computed whole
on one rank, and float addition is not associative -- enough to fork a long
chain-of-thought. So the bar is result equivalence, not token identity.

Measured: on 30 GSM8K items, ADP with either expert TP or
moe_expert_parallel_size=2 reproduces the TP-only answer and correct/incorrect
flag on every item (accuracy 0.9667, zero score flips). On 30 MMMU items paired
against a TP run at identical runtime settings, per-item results do move -- 6
score flips, 4 of them in ADP's favour, net +2 items or 0.76 sigma, which n=30
cannot distinguish from noise.

The MMMU pairing needed its own control first. Two pure-TP runs differing only
in free_gpu_memory_fraction are byte-identical (0/30), but dropping
max_batch_size from 8 to 1 moves 6 scores on its own -- so the arms were held
at the same max_batch_size and only the memory fraction was allowed to differ,
which is what makes an ADP-vs-TP comparison possible at all.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

The entry had grown to ten sentences and 2116 characters -- five times its
neighbours -- because it explained mechanism rather than support. Every other
footnote in the table states what works, what does not, and what a caller must
pass; none explains how it is implemented.

Removed: the attention-DP paragraph and its measured per-rank memory figures
and GSM8K/MMMU per-item results (attention DP is not Inkling-specific, and
`features/parallel-strategy.md` already states that it replicates every GEMM
weight per rank); the fused-MoE factory and CUTLASS scale-remap description;
the Triton decode kernel's reliance on `InklingAttentionMetadata`; the
short-convolution rationale for KV reuse being off; the `ONESHOT` all-reduce
mitigation, which exposes no knob a caller can set; and the per-benchmark
accuracy-gating breakdown, which is test coverage rather than support status.

Kept: what the checkpoint is, the reasoning-parser flag, the two constraints
enforced at load, and the unsupported list. The table columns and the code
comments still carry the rest.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…ents

CUDA graph capture is broken with expert parallelism enabled, so the feature
matrices claim more than holds. Move the column to No for both Inkling rows and
say in the footnote which configuration works, so the entry can go back to Yes
once the expert-parallel case is fixed rather than being rediscovered.

Also trims the comments added with the Inkling-Small references and test
classes down to what the reader cannot get from the code beside them.

Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>